Intelligently identifying and presenting digital documents

ABSTRACT

One or more embodiments of a document organization system quickly and conveniently provide digital documents to a user on a client device based on a physical object. In particular, the document organization system can receive an image of a physical document and an identifier from a first client device, identify digital documents that match the physical document, and provide the matching digital documents to a second client device, which displays the identifier. In another embodiment, the document organization system allows a user to bind digital documents to a physical object and later recall the digital documents using the physical object. In addition, the document organization system can store and recall the layout arrangement of digital documents on a client device when binding and recalling the digital documents to the physical object.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a division of U.S. application Ser. No.15/086,912, filed Mar. 31, 2016. The aforementioned application ishereby incorporated by reference in its entirety.

BACKGROUND

Many users interact with both digital and physical documents on a dailybasis. In many instances, however, as computers improve and technologyadvances, users are shifting away from physical space to digital space.For example, photographers almost exclusively work in digital space whencapturing and editing pictures. As another example, many offices aregoing “paperless” by eliminating the need to store paper copies offiles. Indeed, working with digital documents provides users with aconvenience, collaboration, and accessibility not provided by physicaldocuments. As part of the transition from physical space to digitalspace, however, certain disadvantages have arisen.

To illustrate, the human mind excels at understanding spatialconnections in the physical world, especially connections that arevisually apparent. For example, a user who has various stacks of filesand folders in his or her office, when asked for the location of aparticular file, can quickly and accurately locate the exact location ofthe file. The same user, however, may struggle to locate a digital copyof the file in the digital realm. As such, the user may resort toexternal sources for help, such as a search function on a computer,which can often be slow and unreliable.

As another example, a user may be working with a number of digitaldocuments on a computing device, for example, as part of a work project.If the user needs to switch to another project, the user often closesthe documents from the first project and opens the documents for theother project. Upon switching back to the first project, the user mustre-locate, re-open, and often re-arrange the documents from the firstproject on one or more display devices. Accordingly, the user is oftenmuch more efficient at storing, locating, and retrieving documents inthe physical realm.

Accordingly, there are a number of considerations to be made in documentmanagement and organization when working with digital documents.

SUMMARY

Embodiments of the present disclosure provide benefits and/or solve oneor more of the foregoing or other problems in the art with systems andmethods for document management and organization when working withdigital documents. For example, one or more embodiments include systemsand methods for providing document matching and retrieval based on animage of a physical document captured by a user using a client device.For instance, a user may capture an image of a paper document, and thesystems and methods disclosed herein can provide the user with a digitalcopy of the document and/or other similar documents. Furthermore, theresulting documents can be intelligently provided to the client by wayof a client device and/or digital workspace based on another imagecaptured by the user that is tied to the target device/workspace.

To briefly illustrate, a document management system can provide anidentifier (e.g., a unique image, a unique QR code) for display to auser on a client device. The system can tie the identifier to the clientdevice and/or a digital workspace presented on the client device. Auser, wanting to locate a digital copy of a physical document (e.g., apaper document, a printed photograph, etc.), uses a second clientdevice, such as a mobile device, to capture an image of the physicaldocument and capture an image of the identifier. Using the image of thephysical document, the document management system can search for relateddigital documents, potentially including an exact digital match of thephysical document. Further, using the image of the identifier, thedocument management system can provide the results of the digitaldocument search to the client device tied to the identifier (e.g., thesystem can provide the search results for presentation in a userinterface, such as a digital workspace, displayed on the client device).

In additional embodiments of the present disclosure, a documentmanagement system can bind digital documents to physical objects. Forexample, a user can identify one or more digital documents and providean image of a physical object to be bound to the identified digitaldocuments. When the user wants to later access the set of documents, theuser provides an image of the physical object (e.g., an image of thephysical object captured using a mobile device). In response, thedocument management system uses the image of the physical object toidentify and provide the bound set of documents to the user. Inaddition, the document management system can save a layout/arrangementof the digital documents when bound to the physical object, and thenprovide the digital documents in the saved layout/arrangement when laterprovided to the user.

Accordingly, the disclosed systems and methods allow users to benefitfrom creating synergies between physical space and digital space. Forexample, the disclosed systems and methods leverage connections betweenphysical and digital objects to facilitate intuitive interactions by theuser with physical objects to perform actions within a digitalworkspace. In other words, the systems and methods can bring theadvantages of working with physical objects to digital documents.

Additional features and advantages of the present disclosure will be setforth in the description that follows, and in part will be obvious fromthe description, or may be learned by the practice of such exemplaryembodiments. The features and advantages of such embodiments may berealized and obtained by way of the instruments and combinationsparticularly pointed out in the appended claims. These and otherfeatures will become more fully apparent from the following descriptionand appended claims, or may be learned by the practice of such exemplaryembodiments as set forth hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which the above-recited and otheradvantages and features of the disclosure can be obtained, a moreparticular description of the disclosure briefly described above will berendered by reference to specific embodiments thereof that areillustrated in the appended drawings. It should be noted that thefigures are not drawn to scale, and that elements of similar structureor function are generally represented by like reference numerals forillustrative purposes throughout the figures. Understanding that thesedrawings depict only typical embodiments of the present disclosure andare not therefore to be considered limiting of its scope, the presentdisclosure will be described and explained with additional specificityand detail using the accompanying drawings.

FIG. 1 illustrates a schematic diagram of a communication environment inaccordance with one or more embodiments described herein;

FIG. 2 illustrates a schematic diagram of a document organization systemin accordance with one or more embodiments described herein;

FIG. 3 illustrates a method-flow diagram showing document organizationsystem interacting with a first client device and a second client deviceto identify and provide digital documents in accordance with one or moreembodiments described herein;

FIGS. 4A-4B illustrate an example workspace environment where documentmatching can occur in accordance with one or more embodiments describedherein;

FIG. 5 illustrates another example workspace environment where documentmatching can occur in accordance with one or more embodiments describedherein;

FIG. 6 illustrates a flowchart of a series of acts in a method forproviding relevant digital documents based on a physical object inaccordance with one or more embodiments;

FIG. 7 illustrates a flowchart of a series of acts in a method forbinding digital documents to a physical object in accordance with one ormore embodiments;

FIG. 8 illustrates a block diagram of a computing device in accordancewith one or more embodiments; and

FIG. 9 illustrates a networking environment of a document organizationsystem in accordance with one or more embodiments.

DETAILED DESCRIPTION

The present disclosure provides one or more embodiments of a documentorganization system that intelligently identifies and provides documentsto a user. As one example, a document management system can provide auser with digital documents based on an image of a paper documentcaptured by the user (e.g., using a mobile device). For example, a userscans a picture of a paper copy of a document, and the documentorganization system provides the user with a digital copy of thedocument and/or other related documents. The document organizationsystem can further provide the user with the resulting digital documentsat a particular client device and/or within a particular digitalworkspace based on an image of a unique identifier received from theuser. In another example, a user provides the document organizationsystem with an image of a physical object and the document organizationsystem provides the user with a set of digital documents previouslyassociated with the physical object.

As mentioned above, in some embodiments, the document organizationsystem provides digital matches of physical documents intelligently on aparticular client device and/or digital workspace. In particular, thedocument organization system provides an identifier to a user on a firstclient device (e.g., causes the first client device to display theidentifier). With a second client device, the user captures an image ofa physical document of interest (e.g., a printed document, a printedimage, etc.) as well as an image of the provided identifier. Using theimage of the physical document, the document organization systemidentifies a digital copy of the physical document and/or other digitaldocuments that are similar to the physical document (e.g., documentsthat include text, an image, or other content matching or similar tocontent included in the physical document). The document organizationsystem then utilizes the image of the identifier to determine a location(e.g., a client device and/or digital workspace) where the resultingdigital documents should be provided. In particular, in this example,the document organization system utilizes the identifier to associatethe resulting digital documents with and provide the resulting digitaldocuments to the first client device and/or cause the first clientdevice to display the resulting digital documents.

As further mentioned above, the document organization system allows auser to bind digital documents to a physical object. For example, a usercan associate one or more digital documents with a physical object, suchas a coffee mug, a pen, a book cover, a sticky note, or any otherphysical object. In particular, the user provides an image of thephysical object and an indication of which digital documents to bind,and the document organization system associates the documents with thephysical object in the image. Subsequently, when the user provides animage of the physical object (e.g., in a stored image or live stream) tothe document organization system, the document organization system canprovide the bound documents to the user. In addition, the documentorganization system can preserve an arrangement of the bound documentsas displayed when bound to the physical object. For instance, if a userhas the documents opened and arranged on the display of a desktop clientdevice, the user can request that the document organization system alsostore the arrangement of the digital documents. When the user laterwants to access the set of documents, the document organization systemprovides the digital documents for display on the desktop client devicein the stored arrangement.

As mentioned above, the disclosed document organization system andcorresponding processes provide users with a number of advantages. Asone advantage, the document organization system allows a user to easilyfind digital copies of physical documents. As another advantage, theuser can spatially associate various projects or assignments, eachhaving an associated set of documents, with different physical objects.Then, using the spatial connection, the user can easily recall andaccess various sets of digital documents using the correspondingphysical objects.

As used herein, the term “digital document” refers to any digitalversion of an electronic document also capable of being printed orreproduced in physical form. For example, a digital document can be adigital document file having one of a variety of file formats, such asfile formats associated with word processor, spreadsheet,communications, and/or graphics software applications. Examples ofdigital document include, but are not limited to, HTML files, WORDfiles, pdf files, slideshow presentation files, electronic massages,emails, instant messages, digital images files, etc. Further, digitaldocuments can include content from one or more additional digitaldocuments. For example, a word processing document can include a digitalimage (e.g., a full resolution image).

Additional features and characteristics of one or more embodiments of adocument organization system are described below with respect to theFigures. FIG. 1 illustrates an example embodiment of communicationsystem 100 (or simply, “system 100”) in accordance with one or moreembodiments described herein. As shown, system 100 includes serverdevice 101 comprising document organization system 102, first clientdevice 104 a, second client device 104 b (collectively referred to as“client devices 104”), and document collection 106, that communicatethrough network 108. Although FIG. 1 illustrates a particulararrangement of document organization system 102, client devices 104,document storage 106, and network 108, various additional arrangementsare possible. For example, document organization system 102 may directlycommunicate with document collection 106, bypassing network 108.

The components of system 100, including server device 101, clientdevices 104, and document collection 106 can communicate via network108. Network 108 may include one or more networks and may use one ormore communication platforms or technologies suitable for transmittingdata and/or communication signals. Additional details relating tonetwork 108 are explained below with reference to FIGS. 8 and 9.

Document collection 106 comprises one or more storage devices storingdigital documents or digital copies of documents. The digital documentsstored on document collection 106 can be associated with one of moreusers, such as user 110. For example, document collection 106 can storeone or more documents in a user account belonging to user 110.Additionally, rather than being tied to one or more users, documentcollection 106 can store digital documents openly available to allusers.

As further illustrated in FIG. 1, user 110 may interact with clientdevices 104. User 110 may be an individual (i.e., human user), abusiness, a group, or any other entity. For purposes of explanation,FIG. 1 illustrates only one user, however, it should be understood thatsystem 100 may include any number of users interacting with system 100using one or more corresponding client devices.

Client devices 104 may represent various types of client devices. Forexample, client devices 104 can include a mobile device, such as asmartwatch or other wearable device, a mobile telephone, a smartphone, aPDA, a tablet, a laptop, or another portable device with a camera and/ordisplay. For instance, a user can use a camera on her wearable device tocapture an image and use a virtual-reality device to display documentresults. Furthermore, client devices 104 can include a non-mobiledevice, such as a desktop or server. For instance, first client device104 a may be a mobile client device and second client device 104 b maybe a non-mobile client device. In addition, client devices 104 mayinclude display devices such as televisions, LCD displays, LED displays,monitors, projectors, etc. Generally, as used herein, the term “clientdevice” can include any type of computing device. Additional details andexamples with respect to client devices 104 are discussed below withrespect to FIG. 8.

In general, client devices 104 enable user 110 to interact with documentorganization system 102. According to one example embodiment, user 110uses first client device to capture an image of a physical document andan image of an identifier (displayed, for example, on second clientdevice 104 b). User 110 provides the captured images to documentorganization system 102. Alternatively, the first client deviceprocesses the image and sends metadata of the image (e.g., featurevector information) to one or more servers of document organizationsystem 102. In response, document organization system 102 uses the imageof the physical document to identify a digital copy of the documentand/or any other related digital documents (e.g., from documents storedby document collection 106). Document organization system 102 thenprovides the identified documents to user 110 via second client device104 b. Additional examples of user interactions with documentorganization system 102 are provided below.

FIG. 2 illustrates an example embodiment of document organization system102. As shown, document organization system 102 may include, but is notlimited to, document access manager 212, image analyzer 214, identifiermanager 216, communication manager 218, image matcher 220, objectidentifier 222, and document binder 224. Each of components 212-224 ofdocument organization system 102 may be in communication with oneanother using any suitable communication technologies. In addition,although components of document organization system 102 are shownseparately in FIG. 1, any of components of document organization system102 may be combined into fewer components, such as into a singlecomponent, or divided into more components as may serve a particularembodiment. Further, components of document organization system 102 maybe located on, or implemented by, one or more computing devices, such asone or more server devices.

Each of components of document organization system 102 can comprisesoftware, hardware, or both. For example, each of components of documentorganization system 102 can comprise one or more instructions stored ona computer-readable storage medium and executable by one or moreprocessors of one or more computing devices. When executed by the one ormore processors, the computer-executable instructions cause a computingdevice to perform the methods described herein. Alternatively,components of document organization system 102 can comprise hardware,such as a special purpose processing device to perform a certainfunction or group of functions.

As an overview, in one or more embodiments, document organization system102 intelligently identifies and provides digital documents to usersbased on images captured by the users, as introduced in more detailabove.

As mentioned above, the illustrated example of document organizationsystem 102 includes document access manager 212. Document access manager212 can identify and access various digital documents. For instance,document access manager 212 accesses digital documents stored indocument collection 106 described above with respect to FIG. 1. In oneor more embodiments, document access manager 212 accesses digitaldocuments associated with a user, such as files uploaded by a user todocument collection 106. Alternatively, document access manager 212accesses digital documents otherwise available, such as documentsuploaded by the user's organization (e.g., provided by co-workers,colleagues, etc.), uploaded by other users, and/or made available viathe Internet (e.g., state or federal tax forms, public images, etc.).

In addition, document access manager 212 can classify types of digitaldocuments. For a document classified as a non-image document, documentaccess manager 212 can create one or more images of the document. Forexample, if document access manager 212 identifies a portable documentfile (PDF) that is ten pages in length, for each page of the PDF,document access manager 212 can create an image of the page. Documentaccess manager 212 then associates the PDF page images with the PDF. Ifa non-image document is only a single page, document access manager 212will generally only create a single image for the document.

In connection with document access manager 212 identifying and accessingdigital documents, document organization system 102 includes imageanalyzer 214. In general, image analyzer 214 processes digital documentsaccessed by document access manager 212. More specifically, imageanalyzer 214 analyzes images (e.g., documents classified as images orimages of non-image documents) to determine feature vectors in eachimage. In particular, image analyzer 214 can uses a deep neural networkto compute feature vectors for each image, as further described below.

As mentioned above, document access manager 212 creates an image foreach page of each non-image document. Thus, for a multiple pagedocument, image analyzer 214 will analyze multiple images and determinefeature vectors for each image page of the document. Regardless of thenumber of images associated with a digital document, image analyzer 214can store the determined feature vectors for each image of the digitaldocument for later comparison and use, as described in further detailbelow. Furthermore, in some embodiments, document access manager 212 canidentify, analyze, and store feature vectors for portions of a documentpage. For example, document access manager 212 parses an image page toidentify text and non-text sections and/or features (e.g., figures,graphs, logos, etc.) of the document. Document access manager 212 canstore feature vectors for each identified portion and feature of theimage page. In this manner, if multiple documents share a common image(or feature), document access manager 212 can isolate that image (orfeature) and store the same of similar feature vectors for each of thedigital documents.

Document organization system 102 also includes identifier manager 216.Identifier manager 216 provides an identifier to a client device. Forexample, an identifier can be a barcode, a quick response (QR) code, animage, a sequence of letters and/or numbers, or another other type ofunique identifier. The identifier manager 216 tracks which identifier isprovided to which client device (or user). In this manner, when theidentifier manager 216 receives the identifier or an image of theidentifier from a user, identifier manager 216 can quickly identify towhich client device (or user) the identifier corresponds.

In some embodiments, identifier manager 216 updates and/or refreshes anidentifier based on a time period. For example, identifier manager 216may refresh the identifier sent to a client device each hour, day, week,etc. For instance, upon identifier manager 216 detects that a clientdevice was sent an identifier 24-hours ago, identifier manager 216 canresend or push a new identifier to the client device. Additionally,identifier manager 216 can update the identifier based on a userrequest, the detection of a duplicate identifier, on another type oftrigger. Alternatively, in some embodiments, identifier manager 216 canassociate a client device with a specific identifier (e.g., maintain adatabase linking client devices to identifiers), and when the clientdevice requests an identifier, identifier manager 216 sends thecorresponding identifier to the client device.

Communication manager 218 manages communication between a user anddocument organization system 102. For example, when a user sends animage of a physical document from a client device associated with theuser to document organization system 102, communication manager 218receives the image and provides the image to image analyzer 214 and/orimage matcher 220.

In addition, communication manager 218 provides digital documents,results, and other communications to a user. For example, upon documentaccess manager 212 identifying a matching digital document or relevantdigital documents, communication manager 218 can provide the results tothe user via a client device associated with the user (as identified bythe identifier). Communication manager 218 can also facilitate othercommunications between a user and document organization system 102, asdescribed below.

As mentioned above, document organization system 102 can identifymatching document or relevant digital documents. In particular and asshown in FIG. 2, document organization system 102 includes image matcher220. Image matcher 220 can identify if a digital document matches or isrelevant to an image provided by a user. In addition, image matcher 220can work in conjunction with image analyzer 214 and/or other componentsof document organization system 102.

In one or more embodiments, image matcher 220 compares feature vectorsfrom a received image to stored feature vectors for document collection106. For example, if a user captures an image of a paper document, imagematcher 220 can compare feature vectors of the captured image to storedfeature vectors to determine a matching digital document. In a similarmanner, image matcher 220 can determine if a captured image matchesother relevant documents in a document collection. Additional detail isprovided below regarding comparing and matching images.

Similar to matching images, document organization system 102 canidentify objects in a captured image. In particular, object identifier222 can determine if a captured image includes an object, and if theobject is previously associated with one or more documents in a documentcollection. For example, object identifier 222 can work in connectionwith image analyzer 214 to determine feature vectors for a capturedimage and image matcher 220 to determine if the object matches apreviously received image of the object or images of objects similar tothe object. Further, when object identifier 222 identifies an object,object identifier 222 can work with document binder 224 to bind theobject to one or more digital documents or retrieve one or more bounddocuments.

As mentioned, document binder 224 facilitates binding digital documentsto physical objects or an image of a physical object. In general,document binder 224 binds one or more digital documents to a physicalobject by associating the one or more digital documents with an image ofthe physical object. To illustrate, a user requests to bind or re-bind aset of digital documents with a physical object. In particular, the usercan capture an image of the physical object and identify the digitaldocument(s) to be bound to the physical document, in response to which,document binder 224 binds the identified digital document(s) to thephysical object found within the captured image.

Subsequently, the user may request to access the bound digitaldocuments. In this case, the user provides an image of the physicalobject to document binder 224, and document binder 224 identifies whichdigital documents are currently bound to the physical object. Uponidentifying the bound digital documents, document binder 224 can, viacommunication manager 218, provide the bound digital documents to theuser. Thus, document binder 224 can quickly recall bound documents uponreceiving an indication of a physical object (e.g., from objectidentifier 222) and upon identifying which digital documents are boundto the physical object.

In some embodiments, document binder 224 also binds a client deviceand/or layout arrangement to a physical object. For example, documentbinder 224 may bind a preference or setting that indicates a particulararrangement of the bound digital documents so long as the receivingclient device is able to accommodate the layout. Additionally, and/oralternatively, document binder 224 may bind a preference or setting thatprovides the bound digital documents to a particular client device,unless otherwise specified by the user. Overall, one will appreciatethat document binder 224 can bind, attach, or otherwise associateadditional preferences and/or settings to a physical object and laterrecall as well as apply the additional preferences and/or settings asdescribed above.

FIG. 3 includes a diagram illustrating an example method flow ofdocument organization system 102 interacting with first client device104 a and second client device 104 b to identify digital documentmatches in accordance with one or more embodiments. Documentorganization system 102, first client device 104 a, and second clientdevice 104 b in FIG. 3 can be example embodiments of correspondingcomponents described in connection with FIG. 1. In addition, the sameuser can operate first client device 104 a and second client device 104b. For example, in one embodiment, first client device 104 a is adesktop computer and second client device 104 b is a mobile device, bothassociated with the same user.

As shown in step 302 of FIG. 3, document organization system 102identifies documents. For example, document organization system 102identifies digital documents in a document collection associated with auser or a group of users. Further, in addition to identifying thedocuments, document organization system 102 also accesses the digitaldocuments. For instance, document organization system 102 identifies andaccesses digital documents from a user's document collection.

In step 304, document organization system 102 determines feature vectorsfor each document page. As a note, in pattern recognition and machinelearning, a feature vector is an n-dimensional vector of numericalfeatures that represent an object. Further, in some embodiments, whenrepresenting images, the feature vectors might correspond to the pixelsof an image and when representing texts, the feature vectors mightcorrespond to term occurrence frequencies. In one or more embodiments,the feature vectors might correspond to high-level features that capturemetadata about an image, such as determining that an image looks like aparticular person, object, scene, or other label (e.g., “sunset,”“office desk,” “family,” “President Lincoln,” etc.).

Document organization system 102 can compute feature vectors for imagesof digital documents in the accessed collection of documents. Fordigital documents that are images, document organization system 102 cancompute feature vectors for the images. For digital documents that arenot images, document organization system 102 can convert the documentsinto one or more images. For example, document organization system 102can convert each page of a multi-page document into a document pageimage, before calculating feature vectors for each document page image.

In one or more embodiments, document organization system 102 can employmachine learning via a neural network or a deep neural network todetermine feature vectors for each image. More specifically, rather thanusing machine learning to identify the category to which an imagebelongs, document organization system 102 ends the machine learningprocess one or more iterations before completion. By ending the machinelearning process one or more stages early, the document organizationsystem can represent an image using a set of numbers. For example, insome deep neural networks, the deep neural network represents an imageat the next-to-last stage with roughly 5,000 numbers (or 5,000dimensions). Document organization system 102 stores these number (ordimensions) as feature vectors to numerically represent an image. Asdescribed below, document organization system 102 can compare thefeature vectors from multiple images to determine matches.

In step 306, document organization system 102 provides first clientdevice 104 a with an identifier. For example, document organizationsystem 102 sends a unique image, QR code, or other identifier to firstclient device 104 a. Document organization system 102 can provide theidentifier via an application, web browser, electronic message, or inanother form of communication. In some embodiments, documentorganization system 102 sends the identifier in response to acorresponding request from first client device 104 a, indicating that auser is interested in receiving document search results at first clientdevice 104 a. In one or more embodiments, first client device 104 adisplays the received identifier within an application and/or digitalworkspace running/open on first client device 104 a.

A user captures an image of the identifier on first client device 104 ausing second client device 104 b, as shown in step 308. For example, iffirst client device 104 a is a desktop computer and second client device104 b is a mobile device with a camera, a user can use the camera onsecond client device 104 b to capture an image of the identifierdisplayed on a display device of first client device 104 a. As examples,the user can capture a still image of the identifier with second clientdevice 104 b or just use the camera on second client device 104 b tolive-scan the identifier.

As shown in step 310, the user captures an image of a physical documentusing second client device 104 b. For example, the user can use a cameraof second client device to capture an image of a page of a paperdocument. As a note, steps 308 and 310 can be performed interchangeably.For instance, a user may first capture an image of the physical documentbefore capturing the image of the identifier. Further, the capturedimages can be captured in a variety of sizes and resolutions.

In step 312, second client device 104 b provides the images captured bysecond client device 104 b to document organization system 102. Secondclient device 104 b can transfer the images to document organizationsystem 102 via a wire or wireless network using a variety of networkprotocols. For example, second client device 104 b can post or push thecaptured images to document organization system 102. Second clientdevice 104 b can provide the images together or in separatetransmissions.

Upon receiving the captured images, document organization system 102determines feature vectors for the captured document image, as step 314illustrates. In particular, document organization system 102 receivesthe captured document image and identifier image, analyzes the captureddocument image to determine feature vectors using the methods andprocesses described above, and stores the feature vectors for thecaptured document image. In an alternative embodiment, second clientdevice 104 b computes features of the captured image directly and sendsthe feature vectors to document organization system 102 rather thansending the captured document image. In this case, document organizationsystem 102 skips step 314, as this step is performed by second clientdevice 104 b.

In some embodiments, document organization system 102 associates thecaptured document image with the identifier. For example, upon receivingthe captured images, document organization system 102 recognizes theidentifier as being linked to first client device 104 a. Accordingly, byassociating the captured document image with the identifier, documentorganization system 102 also associates the captured document image withfirst client device 104 a and/or the user.

Step 316 shows document organization system 102 identifying a matchingdigital document or relevant digital documents. More specifically,document organization system 102 can compare the feature vectors of thecaptured document to the feature vectors for the accessed documents in adocument collection to determine whether one or more of the accesseddigital document matches the physical document. In particular, documentorganization system 102 can compare the similarity distance (e.g., L2distance in Euclidean space, binary classification, boundary decisions,and/or support vector machine (SVM) score) between the feature vectorsof the captured image and the feature vectors of a stored image todetermine how close the images match. For example, when using SVM,document organization system 102 can use feature vectors from thecaptured query image as a positive example and use feature vectors fromrandom images as negative examples to train and classify documentorganization system 102. In this manner, document organization system102 can quickly (e.g., less than 100 ms) use SVM to determine similaritydistances between the captured document and the accessed documents in adocument collection. If the images are perfect matches, the similaritydistance will be zero or near zero. As images of stored digitaldocuments diverge from the captured document image, the similaritydistance between the corresponding feature vectors also increases.

In addition to identifying matches, document organization system 102 canidentify near matches or otherwise relevant digital documents from theaccessed documents. For instance, document organization system 102 candetermine that any digital document that has a distance value to thecaptured image below a threshold distance value is a relevant document.For example, a user at a furniture store captures an image of a lamp,document organization system 102 can identify other similar images oflamps from the document collection, even if the lamps in the documentcollection are not exact matches. Alternatively, in some embodiments,document organization system 102 can determine that the top n resultsare relevant documents, ranked from lowest to highest similaritydistances between feature vectors.

As a note, when determining matches, document organization system 102can compare the captured image to each image connected to the accesseddigital documents. For example, if a stored digital document is sevenpages long and has seven document page images, each having its own setof feature vectors, document organization system 102 compares thefeature vectors of the captured document image to the feature vectors ofeach document page image (or parsed portions within each document pageimage). In this manner, if the user provides the third or fourth page ofa physical document as the captured document image, documentorganization system 102 can still find the matching digital documenteven though the matching page is not the first page of the digitaldocument.

In some embodiments, document organization system 102 can compare one ormore captured images from the same source (e.g., multiple pages from thesame physical document, various images of an object, multiple framesfrom a video, etc.) to identify one or more accessed digital documents.For example, document organization system 102 can compare each capturedimage separately and total (and/or average) the aggregate resultstogether to determine one or more relevant accessed digital documents.Alternatively, document organization system 102 can combine (e.g.,average) feature vectors from each captured image together beforeidentifying matching relevant documents from the access digitaldocuments.

In step 318, document organization system 102 provides the digitaldocument results to first client device 104 a in association with theidentifier. Because the identifier is associated with first clientdevice 104 a, document organization system 102 can send the results tofirst client device 104 a. In this manner, if the user is using a clientdevice belonging to a third-party, such as giving a presentation on athird-party client device at a hotel conference, document organizationsystem 102 can provide the identifier to the third-party client device,the user can uses his or her mobile device to capture the identifier andan image of a physical document, and document organization system 102can display the results on the third-party client device, withoutrequiring the user to log in or otherwise provide personal informationvia the third-party client device.

In some embodiments, the user may provide instructions to send theresults to a different target client device, such as second clientdevice 104 b, or another client device associated with the user. Forexample, the user may log on to first client device 104 a and/or secondclient device 104 b and a third client device. When sending in thecaptured images of the identifier and a physical document, the user mayrequest to send the results to the third client device (i.e., requestthe third client device as the target client device). As such, documentorganization system 102 can associated the identifier with the useraccount and provide the results to the third client device.

FIGS. 4A-4B illustrate an example workspace environment 400 (or simply“workspace 400”) to illustrate features and processes disclosed herein.As shown in FIG. 4A, workspace 400 includes first client device 404 a(e.g., a mobile phone device), second client device 404 b (e.g., adesktop computing device), and physical document 406. First clientdevice 404 a and second client device 404 b may be example embodimentsof first client device 104 a and second client device 104 b describedabove. In some cases, workspace 400 represents a user's desk found atthe user's workplace or home. One will appreciate that workspace 400 canrepresent physical workspaces at a variety of locations.

Second client device 404 b in workspace 400 can display identifier 408.While identifier 408 is displayed in the bottom right corner of secondclient device 404 b, the size, location, and/or position of identifierwithin second client device 404 b can vary. Furthermore, the app withinwhich identifier 408 is displayed can also vary. In some embodiments,second client device 404 b displays identifier 408 within a digitalworkspace and/or user interface associated with a corresponding documentmanagement system or service. As mentioned above, identifier 408 can bea unique image, QR code, or another unique type of identifier thatallows the document organization system to associate identifier withsecond client device 404 b. As described below, in some embodiments,identifier 408 is hidden from the view of the user.

A user can operate first client device 404 a and second client device404 b to obtain a digital document that matches physical document 406.For example, a user captures an image of physical document 406 withfirst client device 404 a. The user also captures an image of identifier408 with first client device 404 a. Using first client device 404 a, theuser provides the captured images to a document organization system(e.g., document organization system 102). In response, the documentorganization system identifies matching and/or relevant digitaldocuments and provides the digital documents to the user by way ofsecond client device 404 b.

To illustrate, FIG. 4B shows the second client device 404 b displayingmatching digital documents 410. In particular, matching digitaldocuments displayed on second client device 404 b include electronicdocuments (i.e., Document A and Document B), an image (i.e., Image A), apresentation (i.e., Presentation A), and a video (i.e., Video A). Eachmatching digital document 410 also includes a link to the digitaldocument. In some embodiments, the image of the digital document itselfmay link to the digital document. For example, upon the user selectingthe image of Document A, the document organization system may displayDocument A on second client device 404 b. In various embodiments, thelink may provide a description of the location of the correspondingdocument within a document collection.

As shown in second client device 404 b of FIG. 4B, matching digitaldocument 410 may include exact matches (e.g., Document A) and nearmatches (e.g., Document B and Presentation A). In addition, the documentorganization system can also provide other relevant or similardocuments, such as other digital documents that show pictures of similarbicycles. For example, the document organization system can identify theimage from within the physical document 406 and find digital documentsthat include the image (or near/relevant matches of the image) or thetext of physical document. For instance, the document can match featurevectors from the image in physical document 406 to feature vectors indigital documents, such as matching the image of the bike in physicaldocument 406 to images of the bike in Image A, Presentation A, and VideoA (e.g., matched to a frame in Video A).

In some embodiments, rather than identifying second client device 404 bbased on identifier 408, document organization system can employalternative methods to identify and link first client device 404 a tosecond client device 404 b (e.g., display results on second clientdevice 404 b). As one example, first client device 404 a can usephysical closeness (e.g., Bluetooth, NFC, IR, etc.) to triggerdisplaying results on second client device 404 b. As another example, auser can manually select second client device 404 b, for instance, froma device menu (e.g., a user selects “Conference Room Computer” from adevice menu on first client device 404 a or gives a voice command suchas “show results on my tablet”). In this example, the selectable devicescan be detected within the same network as first client device 404 a,detected by first client device 404 a, and/or associated with the user'saccount. As a further example, the document organization system mayautomatically display results on second client device 404 b (e.g., firstclient device 404 a is paired with second client device 404 b and allresults are automatically displayed on second client device 404 b). Asstill another example, the user may “flick,” “bump,” “nudge,” orotherwise link first client device 404 a and 404 b together.

FIG. 5 illustrates another example workspace environment 500 (or simply“workspace 500”) where document matching can occur in accordance withone or more embodiments. Workspace 500 includes many similar components,devices, and elements as workspace 400 described above. For example,workspace 500 includes first client device 504 a and second clientdevice 504 b. In addition, workspace 500 includes a physical object,such as sticky note 506. Sticky note 506 can have writing on it thatuniquely distinguishes it from other sticky notes that are blank, orthat have different writing on them. In alternative configurations, thephysical object is any other physical object (e.g., a mug, a pen, etc.).

Second client device 504 a displays identifier 508. As described above,identifier 508 can assist the document organization system inidentifying the client device, to which the document organization systemwill provide document results. In addition, second client device 504 balso shows digital documents 512. A user may be using the digitaldocuments 512 and have them arranged in a particular manner. Forexample, a user may open and purposely position the digital documents512 within the display of second client device 504 b.

With the digital documents 512 open, or otherwise selected, a user maybind digital documents 512 to a physical object. For example, the usercan bind digital document 512 to sticky note 506. In particular, theuser can use first client device 504 a to send an image of sticky note506 and identifier 508 to the document organization system. Inconnection with the captured image, the user can provide a request tobind digital documents 512 to sticky note 506. Using the identifier 508,the document organization system can identify digital documents 512 onsecond client device 504 b and bind digital documents 512 to sticky note506. Later, when the user wants to recall digital documents 512, theuser need only provide an image of sticky note 506 to the documentorganization system.

To illustrate by way of example, while working on a first projectinvolving digital documents 512, the user may need to switch to adifferent project involving different documents on second client device504 b. As such, the user binds digital documents 512 to sticky note 506before closing digital documents 512 on second client device 504 b andbefore opening documents related to the other project. For the sake ofthis example, image that sticky note 506 has the words “Project 1”written on it.

When the user wants to return to the first project and digital documents512, the user provides an image of sticky note 506 to the documentorganization system. Specifically, the user provides an image of stickynote 506 with the words “Project 1” to the document organization system.In some instances, the user can provide an indication to recall a set ofbound digital documents when providing the image of sticky note 506. Inresponse, the document organization system recognizes the image ofsticky note 506 including the words “Project 1,” detects that digitaldocuments 512 are bound to sticky note 506, identifies second clientdevice 504 b as the target destination for digital documents 512 basedon identifier 508, and provides digital documents 512 to second clientdevice 504 a. To detect sticky note 506 and/or the words “Project 1”within the captured image provided by the user, the documentorganization system can use machine learning and neural networks asdescribed above.

As an alternative example, the document organization system can retrievemultiple digital documents that are bound to multiple physical objects.For example, a user binds Document A to a first sticky note that reads“Project 1A” and binds Document B to a second sticky note that reads“Project 1B.” Subsequently, the user later provides an image thatincludes both the first sticky note and the second sticky note. Uponreceiving and analyzing the image, the document organization systemdetects both the first sticky note and the second sticky note within theimage. Based on the detection, the document organization system locatesand provides corresponding bound documents Document A and Document B tothe user.

In some embodiments, in addition to binding one or more digitaldocuments to a physical object, the document organization system canalso bind the layout arrangement of the digital documents to thephysical object. For example, as shown in FIG. 5, digital documents 512are arranged in a particular layout. When the user binds digitaldocuments 512 to sticky note 506, the document organization system canstore the layout arrangement. Then, when the user provides sticky note506 to the document organization system to recall digital documents 512,the document organization system can recall the layout arrangement ofdigital documents 512 and display digital documents 512 in the storedlayout arrangement. Further, a user can bind different sets of digitaldocuments to different physical objects. In this manner, the user cancreate a mental mapping of the physical objects, and the documentorganization system can quickly and conveniently provide correspondingdigital documents upon the user's providing images of each physicalobject.

In a number of embodiments, the user may request to recall digitaldocuments 512 to another client device. In providing the digitaldocuments 512 to the other client device, the document organizationsystem can determine if the other client device can display digitaldocuments 512 in the stored layout arrangement, or if the documentorganization system should provide an alternative layout arrangement.For example, if the document organization system is providing digitaldocuments 512 on a client device that has similar display properties assecond client device 504 b (or at least can accommodate the displayrequirements of the stored layout arrangement), the documentorganization system can display digital documents 512 as they arepresented on second client device 504 b in FIG. 5. If the documentorganization system is providing digital documents 512 to a clientdevice that cannot accommodate the display requirements of the storedlayout arrangement, such as a mobile client device, the documentorganization system can provide an alternative layout, such as stackingdigital documents 512 or presenting the digital documents one at a time.

While the present disclosure generally describes using one client deviceto capture images of a physical object and another client device todisplay results, in one or more embodiments, a user can use a singleclient device to both capture images and display results. To illustrate,second client device 504 b in FIG. 5 shows a camera 514. When bindingdigital documents 512 to sticky note 506, the user can use second clientdevice 504 b to capture an image of sticky note 506. Note, because thedocument organization system sends identifier 508 to second clientdevice 504 b, second client device 504 b need to require the user tocapture an image of identifier 508. Rather, second client device 504 bcan just send identifier 508 back to the document organization systemalong with an image of sticky note 506. In these and other cases,identifier 508 can be hidden from the user's view.

Further, in some embodiments, camera 514 on second client device 504 bcaptures live image scans rather than snapping and storing a picture.For example, a user provides a command to second client device 504 b,such as a voice command or a selection of a virtual button, to binddigital documents 512 to sticky note 506. Upon receiving the command,second client device 504 b activates camera 514, which sends a liveimage feed of sticky note 506 to the document organization system. Invarious embodiments, the user can also provide voice commands toidentify document matches of digital documents from a physical document.

In additional embodiments, the document organization system can alsoidentify a user providing a physical gesture, such as pointing, toidentify a physical document or physical object, such as sticky note506. Based on the gesture along with a user command (such as a voicecommand to bind), the document organization system can bind digitaldocument 512 to sticky note 506. Alternatively, in some embodiments,rather that the document organization system, second client device 504 banalyzes the live image feed, detects the gesture indicating sticky note506, and provides an image of sticky note 506 to the documentorganization system.

Likewise, when the user wants to recall digital documents 512, the userprovides a command to activate a live image feed on camera 514. Again,second client device 504 b scans the live image feed and provides thefeed to the document organization system, which detects the userpointing to sticky note 506. Upon detecting the user pointing to stickynote 506, the document organization system can determine that digitaldocuments 512 are bound to sticky note 506. Further, the documentorganization system can provide digital documents 512 to the user tosecond client device 504 b. In this manner, the user can quickly bindand recall one or more digital documents with a single client deviceand/or using physical gestures, such as pointing to physical objects.

In additional embodiments, the document organization system can detectmanipulation of a physical object and apply a modification to acorresponding digital document. For example, the document organizationsystem can detect that a user has highlighted portions of a physicaldocument. As such, the document organization system can add highlightingor similar coloring to the corresponding digital document. As anotherexample, the document organization system detects when a user rotates orfolds a physical object. In response, the document organization systemcan change a property or attribute of a digital document, such as changethe contrast, apply cropping, etc.

In various embodiments, a user can beam digital documents and/or resultsto other client devices. For example, a user may obtain one or morematching digital documents on a mobile client device. The use may useBluetooth, NFC, Wi-Fi, or another technology to beam the digitaldocuments to another client device. In some embodiments, a user canshare an identifier with multiple client devices. Then, when thedocument organization system provides a matching digital document, thedocument organization system can provide the matching digital documentsto multiple client devices that have the identifier at the same time.

FIGS. 1-5, the corresponding text, and the examples, provide a number ofdifferent systems and devices for providing relevant digital documentsbased on a physical object. In addition to the foregoing, one or moreembodiments can also be described in terms of flowcharts comprising actsand steps in a method for accomplishing a particular result. Forexample, FIGS. 6 and 7 illustrate flowcharts of exemplary methods inaccordance with one or more embodiments. The methods described inrelation to FIGS. 6 and 7 may be performed with less or more steps/actsor the steps/acts may be performed in differing orders. Additionally,the steps/acts described herein may be repeated or performed in parallelwith one another or in parallel with different instances of the same orsimilar steps/acts. Furthermore, the thumbnail caching system canperform each of the acts described in relation to FIGS. 6 and 7automatically without user input.

FIG. 6 illustrates a flowchart of an example method 600 for providingrelevant digital documents based on a physical object. While FIG. 6illustrates example steps according to one embodiment, other embodimentsmay omit, add to, reorder, and/or modify any of the steps shown in FIG.6. One or more steps shown in FIG. 6 may be performed by any of thecomponents illustrated in document organization system 102 illustratedin FIG. 2. Further, one or more steps of method 600 may be performed byclient device 104 a, client device 104 b, client device 404 a, clientdevice 404 b, client device 504 a, client device 504 b, server device101, or a combination of both a client device and a server device.

Method 600 includes act 610 of providing an identifier. In particular,act 610 may involve providing a unique identifier 408 for display atfirst client device 404 a. For example, act 610 may include creating aunique identifier. In some embodiments, the identifier is an image, a QRcode, or a sequence of numbers and/or letters. The identifier canuniquely identify a user and/or a client device.

Method 600 further includes act 620 of receiving an image of a document.In particular, act 620 may involve receiving, from a second clientdevice 404 b, an image of a physical document captured by a user. Forexample, act 620 may include receiving an image of a physical object,such as a physical document. In one or more embodiments, the secondclient device is a mobile client device with image capturingcapabilities.

In addition, method 600 includes act 630 of receiving an image of theidentifier. In particular, act 630 may involve receiving, from thesecond client device 404 a, an image of the identifier 408 captured bythe user. For instance, act 630 may include the second client devicecapturing an image of the identifier displayed on the first clientdevice and providing the image of the identifier.

Method 600 also includes act 640 of analyzing the image to identifyrelevant documents. In particular, act 640 may involve analyzing theimage of the physical document to identify one or more digital documentsthat are relevant to the physical document. In addition, step 640 mayinclude accessing a document collection housing the digital documents,where the digital documents each have one or more document pages, andfor each document page, determining one or more feature vectors for eachof the one or more document pages. For example, step 6340 may involveaccessing digital documents uploaded by a user to document collection,uploaded by the user's organization (e.g., provided by co-workers,colleagues, etc.), uploaded by other users, and/or made available viathe Internet (e.g., state or federal tax forms, public images, etc.).

Further, step 640 may involve determining one or more feature vectorsfor image of the physical document, and comparing the one or morefeature vectors of the image of the physical document to the one or morefeature vectors of each of the one or more document pages of theplurality documents in the document collection. In some embodiments, adocument is classified as relevant when the feature vectors of adocument page image of the document has a similarity distance below athreshold distance to the feature vectors of the document image. Invarious embodiments, the document collection comprises text files, imagefiles, and video files.

Method 600 further includes act 650 of providing the relevant documents.In particular, act 650 may involve providing, based on the receivedimage of the identifier 408, the one or more digital documents to thefirst client device 404 a. For example, act 650 may include providingthe identified documents to the client device based on determining thatthe identifier is located on the first client device.

Further, in one or more embodiments, method 600 may include one or moreadditional acts to create and maintain a thumbnail cache. For example,method 600 may include an act of receiving, from the first clientdevice, a selected document from the relevant documents and providing,to the first client device, the selected relevant document. In someembodiments, method 600 may also include an act of receiving, from thefirst client device, instructions to send the selected relevant documentto the second client device.

FIG. 7 illustrates a flowchart of an example method 700 for bindingdigital documents to a physical object. While FIG. 7 illustrates examplesteps according to one embodiment, other embodiments may omit, add to,reorder, and/or modify any of the steps shown in FIG. 7. One or moresteps shown in FIG. 7 may be performed by any of the componentsillustrated in document organization system 102 illustrated in FIG. 2.Further, one or more steps of method 600 may be performed by clientdevice 104 a, client device 104 b, client device 404 a, client device404 b, client device 504 a, client device 504 b, server device 101, or acombination of both a client device and a server device.

Method 700 includes act 710 of identifying documents. In particular, act710 may involve identifying one or more digital documents. For example,act 710 may include receiving an image of the one or more documents froma client device and identifying one or more documents from the documentcollection matching the one or more received images. As another example,act 710 may include receiving a selection of the one or more documentsfrom a client device.

Method 700 further includes act 720 of receiving a first image of anobject. In particular, act 720 may involve receiving a first image of aphysical object captured by a user. In some cases, act 720 may involvereceiving, from a client device, a first image of a physical object. Inone or more embodiments, the client device has image capturingcapabilities. In some embodiments, the physical object is a physicalpaper document or three-dimensional object.

Method 700 additionally includes act 730 of binding the documents to theobject. In particular, act 730 may involve binding the one or moredigital documents to the physical object. For instance, act 730 mayinclude binding the one or more digital documents to a particular clientdevice, where the one or more digital documents bound to the first imageof the object are provided to the particular client device. In anotherinstance, act 730 may include binding the one or more digital documentsto a particular arrangement on a client device, where providing the oneor more digital documents bound to the first image of the objectincludes providing the one or more digital documents to the clientdevice in the particular arrangement. In addition, act 730 may includebinding multiple documents to the first image based on detectingmultiple physical objects within the first image. Further, in someembodiments, act 730 may include receiving a voice command to bind theone or more digital documents to the received first image of the object.

Method 700 further includes act 740 of receiving a second image of theobject. In particular, act 740 may involve receiving a second image ofthe physical object captured by the user. In some cases, act 740 mayinvolve receiving, from a client device, a second image of the physicalobject. In one or more embodiments, the client device has imagecapturing capabilities.

Method 700 also includes act 750 of determining that the second imageincludes the object. In particular, act 750 may involve determining thatthe second image includes the physical object. For example, act 750 mayinclude determining that the second image includes the physical objectby determining that the similarity distance between the feature vectorsfrom the first image of the object and the feature vectors from thesecond image of the object are below a threshold distance value. In someembodiments, method 700 includes an act of using a deep neural networkmachine learning to determine feature vectors for the first image of theobject and the second image of the object.

In addition, method 700 also includes act 760 of retrieving thedocuments. In particular, act 760 may involve retrieving the one or moredigital documents in response to receiving the second image. Forexample, act 760 may include retrieving digital documents that have asimilarity distance between the feature vectors from the first image ofthe object and the feature vectors from the second image of the objectthat are below the threshold distance value.

Method 700 further includes act 770 of providing the documents. Inparticular, act 770 may involve based on determining that the secondimage includes the physical object and based on the binding of the oneor more digital documents to the physical object, the one or moredigital documents to the user. For example, act 760 may includeproviding the documents that are bound to the object to a user via aclient device.

Embodiments of the present disclosure may comprise or utilize a specialpurpose or general-purpose computer including computer hardware, suchas, for example, one or more processors and system memory, as discussedin greater detail below. Embodiments within the scope of the presentdisclosure also include physical and other computer-readable media forcarrying or storing computer-executable instructions and/or datastructures. In particular, one or more of the processes described hereinmay be implemented at least in part as instructions embodied in anon-transitory computer-readable medium and executable by one or morecomputing devices (e.g., any of the media content access devicesdescribed herein). In general, a processor (e.g., a microprocessor)receives instructions, from a non-transitory computer-readable medium,(e.g., a memory, etc.), and executes those instructions, therebyperforming one or more processes, including one or more of the processesdescribed herein.

Computer-readable media can be any available media that can be accessedby a general purpose or special purpose computer system.Computer-readable media that store computer-executable instructions arenon-transitory computer-readable storage media (devices).Computer-readable media that carry computer-executable instructions aretransmission media. Thus, by way of example, and not limitation,embodiments of the disclosure can comprise at least two distinctlydifferent kinds of computer-readable media: non-transitorycomputer-readable storage media (devices) and transmission media.

Non-transitory computer-readable storage media (devices) includes RAM,ROM, EEPROM, CD-ROM, solid state drives (“SSDs”) (e.g., based on RAM),Flash memory, phase-change memory (“PCM”), other types of memory, otheroptical disk storage, magnetic disk storage or other magnetic storagedevices, or any other medium which can be used to store desired programcode means in the form of computer-executable instructions or datastructures and which can be accessed by a general purpose or specialpurpose computer.

A “network” is defined as one or more data links that enable thetransport of electronic data between computer systems and/or modulesand/or other electronic devices. When information is transferred orprovided over a network or another communications connection (eitherhardwired, wireless, or a combination of hardwired or wireless) to acomputer, the computer properly views the connection as a transmissionmedium. Transmissions media can include a network and/or data linkswhich can be used to carry desired program code means in the form ofcomputer-executable instructions or data structures and which can beaccessed by a general purpose or special purpose computer. Combinationsof the above should also be included within the scope ofcomputer-readable media.

Further, upon reaching various computer system components, program codemeans in the form of computer-executable instructions or data structurescan be transferred automatically from transmission media tonon-transitory computer-readable storage media (devices) (or viceversa). For example, computer-executable instructions or data structuresreceived over a network or data link can be buffered in RAM within anetwork interface module (e.g., a “NIC”), and then eventuallytransferred to computer system RAM and/or to less volatile computerstorage media (devices) at a computer system. Thus, it should beunderstood that non-transitory computer-readable storage media (devices)can be included in computer system components that also (or evenprimarily) utilize transmission media.

Computer-executable instructions comprise, for example, instructions anddata which, when executed at a processor, cause a general purposecomputer, special purpose computer, or special purpose processing deviceto perform a certain function or group of functions. In someembodiments, computer-executable instructions are executed on a generalpurpose computer to turn the general purpose computer into a specialpurpose computer implementing elements of the disclosure. The computerexecutable instructions may be, for example, binaries, intermediateformat instructions such as assembly language, or even source code.Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the described features or acts described above.Rather, the described features and acts are disclosed as example formsof implementing the claims.

Those skilled in the art will appreciate that the disclosure may bepracticed in network computing environments with many types of computersystem configurations, including, personal computers, desktop computers,laptop computers, message processors, hand-held devices, multi-processorsystems, microprocessor-based or programmable consumer electronics,network PCs, minicomputers, mainframe computers, mobile telephones,PDAs, tablets, pagers, routers, switches, and the like. The disclosuremay also be practiced in distributed system environments where local andremote computer systems, which are linked (either by hardwired datalinks, wireless data links, or by a combination of hardwired andwireless data links) through a network, both perform tasks. In adistributed system environment, program modules may be located in bothlocal and remote memory storage devices.

Embodiments of the present disclosure can also be implemented in cloudcomputing environments. In this description, “cloud computing” isdefined as a model for enabling on-demand network access to a sharedpool of configurable computing resources. For example, cloud computingcan be employed in the marketplace to offer ubiquitous and convenienton-demand access to the shared pool of configurable computing resources.The shared pool of configurable computing resources can be rapidlyprovisioned via virtualization and released with low management effortor service provider interaction, and then scaled accordingly.

A cloud-computing model can be composed of various characteristics suchas, for example, on-demand self-service, broad network access, resourcepooling, rapid elasticity, measured service, and so forth. Acloud-computing model can also expose various service models, such as,for example, Software as a Service (“SaaS”), Platform as a Service(“PaaS”), and Infrastructure as a Service (“IaaS”). A cloud-computingmodel can also be deployed using different deployment models such asprivate cloud, community cloud, public cloud, hybrid cloud, and soforth. In this description and in the claims, a “cloud-computingenvironment” is an environment in which cloud computing is employed.

FIG. 8 illustrates a block diagram of exemplary computing device 800that may be configured to perform one or more of the processes describedabove. One will appreciate that computing device 800 may representclient devices (e.g., client device 104 a, client device 104 b, clientdevice 404 a, client device 404 b, client device 405 a, 4 client device405 b) and/or service device 101. As shown by FIG. 8, computing device800 can comprise processor 802, memory 804, storage device 806, I/Ointerface 808, and communication interface 810, which may becommunicatively coupled by way of communication infrastructure 812.While an exemplary computing device 800 is shown in FIG. 8, thecomponents illustrated in FIG. 8 are not intended to be limiting.Additional or alternative components may be used in other embodiments.Furthermore, in certain embodiments, computing device 800 can includefewer components than those shown in FIG. 8. Components of computingdevice 800 shown in FIG. 8 will now be described in additional detail.

In particular embodiments, processor 802 includes hardware for executinginstructions, such as those making up a computer program. As an exampleand not by way of limitation, to execute instructions, processor 802 mayretrieve (or fetch) the instructions from an internal register, aninternal cache, memory 804, or storage device 806 and decode and executethem. In particular embodiments, processor 802 may include one or moreinternal caches for data, instructions, or addresses. As an example andnot by way of limitation, processor 802 may include one or moreinstruction caches, one or more data caches, and one or more translationlookaside buffers (TLBs). Instructions in the instruction caches may becopies of instructions in memory 804 or storage 806.

Memory 804 may be used for storing data, metadata, and programs forexecution by the processor(s). Memory 804 may include one or more ofvolatile and non-volatile memories, such as Random Access Memory(“RAM”), Read Only Memory (“ROM”), a solid state disk (“SSD”), Flash,Phase Change Memory (“PCM”), or other types of data storage. Memory 804may be internal or distributed memory.

Storage device 806 includes storage for storing data or instructions. Asan example and not by way of limitation, storage device 806 can comprisea non-transitory storage medium described above. Storage device 806 mayinclude a hard disk drive (HDD), a floppy disk drive, flash memory, anoptical disc, a magneto-optical disc, magnetic tape, or a UniversalSerial Bus (USB) drive or a combination of two or more of these. Storagedevice 806 may include removable or non-removable (or fixed) media,where appropriate. Storage device 806 may be internal or external tocomputing device 800. In particular embodiments, storage device 806 isnon-volatile, solid-state memory. In other embodiments, Storage device806 includes read-only memory (ROM). Where appropriate, this ROM may bemask programmed ROM, programmable ROM (PROM), erasable PROM (EPROM),electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM),or flash memory or a combination of two or more of these.

I/O interface 808 allows a user to provide input to, receive outputfrom, and otherwise transfer data to and receive data from computingdevice 800. I/O interface 808 may include a mouse, a keypad or akeyboard, a touch screen, a camera, an optical scanner, networkinterface, modem, other known I/O devices or a combination of such I/Ointerfaces. I/O interface 808 may include one or more devices forpresenting output to a user, including, but not limited to, a graphicsengine, a display (e.g., a display screen), one or more output drivers(e.g., display drivers), one or more audio speakers, and one or moreaudio drivers. In certain embodiments, I/O interface 808 is configuredto provide graphical data to a display for presentation to a user. Thegraphical data may be representative of one or more graphical userinterfaces and/or any other graphical content as may serve a particularimplementation.

Communication interface 810 can include hardware, software, or both. Inany event, communication interface 810 can provide one or moreinterfaces for communication (such as, for example, packet-basedcommunication) between computing device 800 and one or more othercomputing devices or networks. As an example and not by way oflimitation, communication interface 810 may include a network interfacecontroller (NIC) or network adapter for communicating with an Ethernetor other wire-based network or a wireless NIC (WNIC) or wireless adapterfor communicating with a wireless network, such as a WI-FI.

Additionally or alternatively, communication interface 810 mayfacilitate communications with an ad hoc network, a personal areanetwork (PAN), a local area network (LAN), a wide area network (WAN), ametropolitan area network (MAN), or one or more portions of the Internetor a combination of two or more of these. One or more portions of one ormore of these networks may be wired or wireless. As an example,communication interface 810 may facilitate communications with awireless PAN (WPAN) (such as, for example, a BLUETOOTH WPAN), a WI-FInetwork, a WI-MAX network, a cellular telephone network (such as, forexample, a Global System for Mobile Communications (GSM) network), orother suitable wireless network or a combination thereof.

Additionally, communication interface 810 may facilitate communicationsvarious communication protocols. Examples of communication protocolsthat may be used include, but are not limited to, data transmissionmedia, communications devices, Transmission Control Protocol (“TCP”),Internet Protocol (“IP”), File Transfer Protocol (“FTP”), Telnet,Hypertext Transfer Protocol (“HTTP”), Hypertext Transfer Protocol Secure(“HTTPS”), Session Initiation Protocol (“SIP”), Simple Object AccessProtocol (“SOAP”), Extensible Mark-up Language (“XML”) and variationsthereof, Simple Mail Transfer Protocol (“SMTP”), Real-Time TransportProtocol (“RTP”), User Datagram Protocol (“UDP”), Global System forMobile Communications (“GSM”) technologies, Code Division MultipleAccess (“CDMA”) technologies, Time Division Multiple Access (“TDMA”)technologies, Short Message Service (“SMS”), Multimedia Message Service(“MMS”), radio frequency (“RF”) signaling technologies, Long TermEvolution (“LTE”) technologies, wireless communication technologies,in-band and out-of-band signaling technologies, and other suitablecommunications networks and technologies.

Communication infrastructure 812 may include hardware, software, or boththat couples components of computing device 800 to each other. As anexample and not by way of limitation, communication infrastructure 812may include an Accelerated Graphics Port (AGP) or other graphics bus, anEnhanced Industry Standard Architecture (EISA) bus, a front-side bus(FSB), a HYPERTRANSPORT (HT) interconnect, an Industry StandardArchitecture (ISA) bus, an INFINIBAND interconnect, a low-pin-count(LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, aPeripheral Component Interconnect (PCI) bus, a PCI-Express (PCIe) bus, aserial advanced technology attachment (SATA) bus, a Video ElectronicsStandards Association local (VLB) bus, or another suitable bus or acombination thereof.

FIG. 9 is a schematic diagram illustrating an environment 900 withinwhich one or more embodiments of document organization system 902 can beimplemented. Document organization system 902 may generate, store,manage, receive, and send digital content associated with digitaldocuments. For example, document organization system 902 may send andreceive digital documents to and from client devices 906 by way ofnetwork 904.

In particular, document organization system 902 can manage synchronizingdigital documents across multiple client devices 906 associated with oneor more users. For example, a user may capture a physical documentand/or physical object using client device 906. The documentorganization system 902 can cause client device 906 to send the capturedimages to document organization system 902. Document organization system902 can then identify matching and/or relevant digital documents on oneor more additional computing devices.

Client device 906 may be a desktop computer, a laptop computer, a tabletcomputer, a personal digital assistant (PDA), an in- or out-of-carnavigation system, a handheld device, a smart phone or other cellular ormobile phone, or a mobile gaming device, other mobile device, or othersuitable computing devices. Client device 906 may execute one or moreclient applications, such as a web browser (e.g., Microsoft WindowsInternet Explorer, Mozilla Firefox, Apple Safari, Google Chrome, Opera,etc.) or a native or special-purpose client application (e.g., Facebookfor iPhone or iPad, Facebook for Android, etc.), to access and viewcontent over network 904.

Network 904 may represent a network or collection of networks (such asthe Internet, a corporate intranet, a virtual private network (VPN), alocal area network (LAN), a wireless local area network (WLAN), acellular network, a wide area network (WAN), a metropolitan area network(MAN), or a combination of two or more such networks) over which clientdevices 906 may access document organization system 902.

In the foregoing specification, the invention has been described withreference to specific exemplary embodiments thereof. Various embodimentsand aspects of the invention(s) are described with reference to detailsdiscussed herein, and the accompanying drawings illustrate the variousembodiments. The description above and drawings are illustrative of theinvention and are not to be construed as limiting the invention.Numerous specific details are described to provide a thoroughunderstanding of various embodiments of the present invention.

In the foregoing specification, the present disclosure has beendescribed with reference to specific exemplary embodiments thereof.Various embodiments and aspects of the present disclosure(s) aredescribed with reference to details discussed herein, and theaccompanying drawings illustrate the various embodiments. Thedescription above and drawings are illustrative of the disclosure andare not to be construed as limiting the disclosure. Numerous specificdetails are described to provide a thorough understanding of variousembodiments of the present disclosure.

The present disclosure may be embodied in other specific forms withoutdeparting from its spirit or essential characteristics. The describedembodiments are to be considered in all respects only as illustrativeand not restrictive. For example, the methods described herein may beperformed with less or more steps/acts or the steps/acts may beperformed in differing orders. Additionally, the steps/acts describedherein may be repeated or performed in parallel with one another or inparallel with different instances of the same or similar steps/acts. Thescope of the present application is, therefore, indicated by theappended claims rather than by the foregoing description. All changesthat come within the meaning and range of equivalency of the claims areto be embraced within their scope.

What is claimed is:
 1. A computer-implemented method comprising:identifying one or more digital documents; receiving a first image of aphysical object captured by a user; binding the one or more digitaldocuments to the first image of the physical object captured by theuser; receiving a second image of the physical object captured by theuser; determining, by at least one processor, that the second imageincludes the physical object; and providing, in response to determiningthat the second image includes the physical object, the one or moredigital documents to the user.
 2. The method of claim 1, furthercomprising using a deep neural network machine learning process todetermine first feature vectors for the first image of the physicalobject and second feature vectors for the second image of the physicalobject.
 3. The method of claim 2, further comprising determining that asimilarity distance between the first feature vectors for the firstimage of the physical object and the second feature vectors for thesecond image of the physical object are below a threshold similaritydistance value based on the similarity distance being below thethreshold similarity distance value.
 4. The method of claim 3, whereinthe physical object is a physical paper document or a three-dimensionalobject.
 5. The method of claim 1, further comprising binding multipledigital documents to the first image based on multiple physical objectsdetected within the first image.
 6. The method of claim 1, furthercomprising binding the one or more digital documents to a client device,and wherein providing the one or more digital documents to the usercomprises providing the one or more digital documents to the clientdevice.
 7. The method of claim 1, further comprising binding the one ormore digital documents to a particular arrangement of the one or moredigital documents on a client device, wherein providing the one or moredigital documents to the user comprises providing the one or moredigital documents in the particular arrangement.
 8. The method of claim1, further comprising: accessing a document collection including aplurality of the documents each having one or more document pages; anddetermining, for each document in the document collection, one or morefeature vectors for each of the one or more document pages.
 9. Themethod of claim 1, further comprising receiving a voice command to bindthe one or more digital documents to the physical object.
 10. A systemcomprising: at least one processor; and at least one non-transitorycomputer-readable storage medium storing instructions that, whenexecuted by the at least one processor, cause the system to: identifyone or more digital documents; receive a first image of a physicalobject captured by a user; bind the one or more digital documents to thefirst image of the physical object captured by the user; receive asecond image of the physical object captured by the user; determine thatthe second image includes the physical object; and provide, in responseto determining that the second image includes the physical object, theone or more digital documents to the user.
 11. The system of claim 10,further comprising instructions that, when executed by the at least oneprocessor, cause the system to use a deep neural network machinelearning process to determine first feature vectors for the first imageof the physical object and second feature vectors for the second imageof the physical object.
 12. The system of claim 11, further comprisinginstructions that, when executed by the at least one processor, causethe system to determine that a similarity distance between the firstfeature vectors for the first image of the physical object and thesecond feature vectors for the second image of the physical object arebelow a threshold similarity distance value.
 13. The system of claim 12,wherein determining that the second image includes the physical objectis based on the similarity distance being below the threshold similaritydistance value.
 14. The system of claim 11, further comprisinginstructions that, when executed by the at least one processor, causethe system to determine one or more feature vectors for each page of theone or more digital documents.
 15. The system of claim 11, furthercomprising instructions that, when executed by the at least oneprocessor, cause the system to determine feature vectors for a portionof a page of the one or more digital documents.
 16. The system of claim11, wherein: binding the one or more digital documents to the physicalobject comprises binding the one or more digital documents to aparticular arrangement of the one or more digital documents on a clientdevice; and providing the one or more digital documents comprisesproviding the one or more digital documents in the particulararrangement.
 17. The system of claim 10, wherein the physical object isa three-dimensional object.
 18. A non-transitory computer-readablemedium storing instructions thereon that, when executed by at least oneprocessor, cause a computer system to: identify one or more digitaldocuments; receive a first image of a physical object captured by auser; bind the one or more digital documents to the first image of thephysical object captured by the user; receive a second image of thephysical object captured by the user; determine that the second imageincludes the physical object; and provide, in response to determiningthat the second image includes the physical object, the one or moredigital documents to the user.
 19. The non-transitory computer-readablemedium of claim 18, wherein: binding the one or more digital documentsto the physical object comprises binding the one or more digitaldocuments to a particular arrangement of the one or digital documents;and providing the one or more digital documents comprises providing theone or more digital documents to a client device in the particulararrangement.
 20. The non-transitory computer-readable medium of claim18, instructions that, when executed by the at least one processor,cause the computer system to: access a document collection including aplurality of the documents each having one or more document pages; anddetermine, for each document in the document collection, one or morefeature vectors for each of the one or more document pages.